Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Multiple fixes for async iterators #23901

Closed
wants to merge 2 commits into from

Conversation

mcollina
Copy link
Member

Fixes: #23890
Fixes: #23891

Checklist
  • make -j4 test (UNIX), or vcbuild test (Windows) passes
  • tests and/or benchmarks are included
  • commit message follows commit guidelines

@mcollina mcollina requested review from TimothyGu and devsnek October 26, 2018 11:41
@mcollina
Copy link
Member Author

lib/internal/streams/async_iterator.js Show resolved Hide resolved
@@ -150,7 +153,7 @@ const createReadableStreamAsyncIterator = (stream) => {
});

finished(stream, (err) => {
if (err) {
if (err && err.code !== 'ERR_STREAM_PREMATURE_CLOSE') {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this the only way to fix it? I would’ve expected us to perhaps do something with end-of-stream rather than selectively ignoring errors.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

end-of-stream treats non-ending (destroy) streams that close early as an error condition. I think it is right in doing so.

However this does not play well with async iterators as you pointed out. We had a different logic before that did the same but was less explicit about it: I like this one better.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

end-of-stream treats non-ending (destroy) streams that close early as an error condition. I think it is right in doing so.

Can you elaborate? It sounds to me that destroyed streams (with null as error value) should be treated no differently as any other stream that ended. I.e., it's reasonable to believe a stream can be destroyed whenever.

In fact, I would say that the following test case added in #23785 should in fact finish gracefully with an { done: true, value: undefined } rather than throwing an error:

await (async function() {
console.log('.next() on destroyed stream');
const readable = new Readable({
read() {
// no-op
}
});
readable.destroy();
try {
await readable[Symbol.asyncIterator]().next();
} catch (e) {
assert.strictEqual(e.code, 'ERR_STREAM_PREMATURE_CLOSE');
}
})();

We see precedent for invalidated iterator still operating gracefully in ECMA-262 iterators:

const array = [0, 1, 2, 3];
const iterated = [];
for (const a of array) {
  iterated.push(a);
  if (a === 1) {
    array.splice(0);
  }
  // loop ends immediately.
}
// iterated is [0, 1]

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

First, end-of-stream is one of the most downloaded utilities on npm, and keeping the API and behavior similar has been a goal. Second, from a consumer perspective a stream that was ended or destroyed is significantly different. That could have easily been a flag as an argument. I think it errors because in the context of pipeline/pump, in fact it is an error: the pipeline did not complete correctly but was interrupted (as an example, gzipping a file).

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I’ll check again that test asap, it does not sound correct with this change.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Okay, as long as async iteration isn't affected it works for me.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@TimothyGu the test was botched anyway, and the catch clause was never executed in this case. I've fixed it.

@mcollina mcollina force-pushed the multiple-fix-async-iterators branch from 2566bab to 3d41222 Compare October 27, 2018 13:22
@mcollina
Copy link
Member Author

@targos
Copy link
Member

targos commented Oct 28, 2018

Landed in 398418d and b1e1fe4

@targos targos closed this Oct 28, 2018
@targos targos added stream Issues and PRs related to the stream subsystem. promises Issues and PRs related to ECMAScript promises. labels Oct 28, 2018
targos pushed a commit that referenced this pull request Oct 28, 2018
Fixes: #23891

PR-URL: #23901
Reviewed-By: Colin Ihrig <cjihrig@gmail.com>
Reviewed-By: James M Snell <jasnell@gmail.com>
targos pushed a commit that referenced this pull request Oct 28, 2018
Fixes: #23890

PR-URL: #23901
Reviewed-By: Colin Ihrig <cjihrig@gmail.com>
Reviewed-By: James M Snell <jasnell@gmail.com>
targos pushed a commit that referenced this pull request Oct 28, 2018
Fixes: #23891

PR-URL: #23901
Reviewed-By: Colin Ihrig <cjihrig@gmail.com>
Reviewed-By: James M Snell <jasnell@gmail.com>
targos pushed a commit that referenced this pull request Oct 28, 2018
Fixes: #23890

PR-URL: #23901
Reviewed-By: Colin Ihrig <cjihrig@gmail.com>
Reviewed-By: James M Snell <jasnell@gmail.com>
targos pushed a commit that referenced this pull request Nov 1, 2018
Fixes: #23891

PR-URL: #23901
Reviewed-By: Colin Ihrig <cjihrig@gmail.com>
Reviewed-By: James M Snell <jasnell@gmail.com>
targos pushed a commit that referenced this pull request Nov 1, 2018
Fixes: #23890

PR-URL: #23901
Reviewed-By: Colin Ihrig <cjihrig@gmail.com>
Reviewed-By: James M Snell <jasnell@gmail.com>
@MylesBorins
Copy link
Contributor

landed into 10.x with #23785

opted to not land in 8.x (I don't believe the async iteration work is on that release stream)

@mcollina
Copy link
Member Author

mcollina commented Nov 26, 2018 via email

MylesBorins pushed a commit that referenced this pull request Nov 26, 2018
Fixes: #23891

PR-URL: #23901
Reviewed-By: Colin Ihrig <cjihrig@gmail.com>
Reviewed-By: James M Snell <jasnell@gmail.com>
MylesBorins pushed a commit that referenced this pull request Nov 26, 2018
Fixes: #23890

PR-URL: #23901
Reviewed-By: Colin Ihrig <cjihrig@gmail.com>
Reviewed-By: James M Snell <jasnell@gmail.com>
@codebytere codebytere mentioned this pull request Nov 27, 2018
rvagg pushed a commit that referenced this pull request Nov 28, 2018
Fixes: #23891

PR-URL: #23901
Reviewed-By: Colin Ihrig <cjihrig@gmail.com>
Reviewed-By: James M Snell <jasnell@gmail.com>
rvagg pushed a commit that referenced this pull request Nov 28, 2018
Fixes: #23890

PR-URL: #23901
Reviewed-By: Colin Ihrig <cjihrig@gmail.com>
Reviewed-By: James M Snell <jasnell@gmail.com>
MylesBorins pushed a commit that referenced this pull request Nov 29, 2018
Fixes: #23891

PR-URL: #23901
Reviewed-By: Colin Ihrig <cjihrig@gmail.com>
Reviewed-By: James M Snell <jasnell@gmail.com>
MylesBorins pushed a commit that referenced this pull request Nov 29, 2018
Fixes: #23890

PR-URL: #23901
Reviewed-By: Colin Ihrig <cjihrig@gmail.com>
Reviewed-By: James M Snell <jasnell@gmail.com>
@codebytere codebytere mentioned this pull request Nov 29, 2018
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
promises Issues and PRs related to ECMAScript promises. stream Issues and PRs related to the stream subsystem.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

7 participants